A Dependency-based Method for Evaluating Broad-Coverage Parsers

A b s t r a c t Wi th th t emergence of broad-coverage parsers, quan­ titative evaluation of parsers becomes increasingly more important We propose a dependency-based method for evaluating broad-coverage parsers The method offers several advantages over previous methods that are based on phrase boundaries The error count score WL propose here is not only more mtui t ivt lv meaningful than other scores but also more relevant lo semantic interpreta­ tion We will also present an algorithm for transform­ ing constituency trees into dependency trees so thai the (.valuation method is applicable to both dependency and constituency grammars Finally we discuss a set of op erations for modifying dependency trees that can be used lo eliminate inconsequential differences among different parse trees and allow us to selectively evaluate different aspects of a parser